Methods and devices for three-dimensional image reconstruction using single-view projection image

ABSTRACT

The disclosure provides a method, device and a computer-readable medium for performing three-dimensional blood vessel reconstruction. The device includes an interface configured to receive a single-view two-dimensional image of a blood vessel of a patient, where the single-view two-dimensional image is a projection image acquired in a predetermined projection direction. The device further includes a processor configured to estimate three-dimensional information of the blood vessel from the single-view two-dimensional image using an inference model, and reconstruct a three-dimensional model of the blood vessel based on the three-dimensional information.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the benefits of priority to U.S.Provisional Application No. 63/248,999, filed Sep. 27, 2021, the contentof which is incorporated herein by reference in its entirety. Thepresent application also relates to U.S. application Ser. No.17/497,980, filed Oct. 11, 2021, the content of which is alsoincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to three-dimensional imagereconstruction. More specifically, the present disclosure relates tomethods and systems for performing three-dimensional imagereconstruction using a single-view projection image.

BACKGROUND

Two-dimensional (2D) X-ray angiographic images provide valuablegeometric information on vascular structures for diagnoses of variousvascular diseases, such as coronary artery diseases and cerebraldiseases. After a contrast agent (usually an x-ray opaque material, suchas iodine) is injected into the vessel, the image contrast of the vesselregions is generally enhanced. Three-dimensional (3D) vascular treereconstruction using the 2D projection images is often beneficial toreveal the true 3D measurements, including diameters, curvatures andlengths, of various vessel segments of interests, for further functionalassessments of the targeted vascular regions.

Although vessels are usually 3D tortuous tube-like object, anangiographic image only provides a projection view from a certain angle.Traditionally, multiple angiographic images projected from differentangles are required in order to reconstruct the 3D model of the targetvessel. One technical challenge presented by such methods is theforeshortening issue. The vessel lengths are slightly different whenviewed from different angles due to the nature of the projectionimaging, causing foreshortening. Generally, foreshortening may bereduced by avoiding using images containing pronounced foreshorteningvessel segments (represented with darker intensity) for 3Dreconstruction. However, at least some level of foreshorteningfrequently occurs due to the curved geometrical nature of vessels anddue to physiological motion of the patient during the imaging process(e.g., due to respiratory motion and cardiac motion).

Moreover, the existing 3D reconstruction method not only requiremulti-view projection images, but also the projection angles need tomeet certain criteria, such as minimum angle difference, in order forthese multi-view reconstruction algorithms to work satisfactorily. Thismakes the task of reconstructing 3D vessel model challenging, and notalways attainable.

Embodiments of the disclosure address the above problems by systems andmethods for improved three-dimensional image reconstructions.

SUMMARY

Embodiments of the present disclosure include computer-implementedmethods and devices for performing three-dimensional blood vesselreconstruction using a single-view projection image.

In one aspect, the disclosure is further directed to a device forperforming three-dimensional blood vessel reconstruction. The deviceincludes an interface configured to receive a single-viewtwo-dimensional image of a blood vessel of a patient, where thesingle-view two-dimensional image is a projection image acquired in apredetermined projection direction. The device further includes aprocessor configured to estimate three-dimensional information of theblood vessel from the single-view two-dimensional image using aninference model, and reconstruct a three-dimensional model of the bloodvessel based on the three-dimensional information.

In another aspect, the disclosure is directed to a computer-implementedmethod for performing three-dimensional image reconstruction. Thecomputer-implemented method includes receiving a single-viewtwo-dimensional image of a patient, where the single-viewtwo-dimensional image is a projection image acquired in a predeterminedprojection direction. The method further includes estimating, by aprocessor, three-dimensional information from the single-viewtwo-dimensional image using an inference model, and reconstructing, bythe processor, a three-dimensional model based on the three-dimensionalinformation.

In yet another embodiment, the disclosure is directed to anon-transitory computer-readable medium, having instructions storedthereon. The instructions, when executed by a processor, perform amethod for performing three-dimensional image reconstruction. The methodincludes receiving a single-view two-dimensional image of a patient,where the single-view two-dimensional image is a projection imageacquired in a predetermined projection direction. The method furtherincludes estimating three-dimensional information from the single-viewtwo-dimensional image using an inference model, and reconstructing athree-dimensional model based on the three-dimensional information.

Capable of using only one projection view to perform the initialreconstruction of a 3D vessel model, the disclosed method and device canreduce the amount of radiation exposure for doctor and patients. Theyalso relax requirement for obtaining 3D vessel reconstruction, as itremoves the stringent requirements for traditional multi-viewreconstruction algorithm, which requires at least two projection viewsfrom sufficiently different angles that both show the target vesselclearly without overlapping with other nearby vessels. Reconstructingfrom a single-view projection image is also faster compared tomulti-view reconstruction, which requires finding correspondence pointsamong different views.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments, and together with thedescription and claims, serve to explain the disclosed embodiments. Whenappropriate, the same reference numbers are used throughout the drawingsto refer to the same or like parts. Such embodiments are demonstrativeand not intended to be exhaustive or exclusive embodiments of thepresent method, device, or non-transitory computer readable mediumhaving instructions thereon for implementing the method.

FIG. 1 illustrates angiographic imaging through projection of a bloodvessel, according to certain embodiments of the present disclosure.

FIG. 2 illustrates a schematic diagram of an exemplary imagereconstruction system, according to certain embodiments of the presentdisclosure.

FIG. 3 illustrates a schematic diagram of an image processing device,according to certain embodiments of the present disclosure.

FIG. 4 illustrates an exemplary three-dimensional image reconstructionframework for performing a single-view three-dimensional reconstruction,according to certain embodiments of the present disclosure.

FIG. 5A illustrates an exemplary three-dimensional image reconstructionframework for performing a single-view three-dimensional reconstructionusing a depth-based approach, according to an embodiment of the presentdisclosure.

FIG. 5B illustrates an exemplary three-dimensional image reconstructionframework for performing a single-view three-dimensional reconstructionusing a model-based approach, according to an embodiment of the presentdisclosure.

FIG. 6 shows a flowchart of an exemplary process for performing athree-dimensional image reconstruction, according to certain embodimentsof the present disclosure.

FIG. 7 illustrates an exemplary process for training a three-dimensionalimage reconstruction framework, according to certain embodiments of thepresent disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to the exemplary embodiments,examples of which are illustrated in the accompanying drawings.

The present disclosure provides a 3D image reconstruction method thatcan reconstruct a 3D model of an object (e.g., a blood vessel) from asingle-view 2D projection image captured of that object, without therequirement of multi-view projection images. The disclosed imagereconstruction method may first estimate three-dimensional informationfrom the single-view 2D projection image (e.g., an X-ray angiographicimage) using an inference model. For example, in a depth-based approach,the three-dimensional information estimated by the inference model maybe depth information of key points or dense points of the object. In amodel-based approach, the three-dimensional information may be modelparameters that characterize the 3D model of the object, including,e.g., shape parameters and pose parameters. The disclosed imagereconstruction method then reconstructs a three-dimensional image basedon the three-dimensional information. For example, in the depth-basedapproach, the 3D image of the object is reconstructed using thecoordinates in the transverse plane (e.g., the projection plane) and thedepth information estimated by the inference model. In the model-basedapproach, the 3D model of the object can be reconstructed using theshape parameters and pose parameters. The disclosed diagnostic imagereconstruction method thus can reconstruct a 3D model of the object fromjust a single-view projection image.

The disclosed image reconstruction method takes advantage of the physicsbehind the formation of a projection image. For example, FIG. 1illustrates angiographic imaging through projection of a blood vessel,according to certain embodiments of the present disclosure. As shownFIG. 1 , vessel 101 is a 3D object in a patient. For example, vessel 101may be a blood vessel, such as a coronary artery vessel, a cerebralartery, a blood vessel in the eye, or a vein, etc. It is contemplatedthat vessel 101 may be other types of vessel or any tree-structured 3Dobject, such as an air pathway in the lung. X-ray beams 102 areprojected on vessel 101 to form an angiographic projection image 103 onthe projection plane. For example, a contrast agent is injected intovessel 101, followed by projecting X-ray beams 102 to vessel 101 using aC-arm X-ray scanner.

Based on angiography X-ray physics, each pixel's intensity value isproportional to the accumulation of exponential attenuation along thex-ray traveling path along the material. For example, as shown in FIG. 1, the traveling distances X_(c) (X_(c1), X_(c2), . . . , and X_(cn)) atmultiple positions of the blood vessel in the projection direction ofthe blood vessel dictate the attenuation of the X-ray at thoserespective positions of the blood vessel. For simplicity, assume thereare two materials, one for the contrast agent inside vessel lumen andanother for the other organ tissues. The mathematical relationship canbe written as Equation 1.I∝exp(−(λ_(c) X _(c)+λ_(o) X _(o)))  (Eq. 1)where λ_(c) and λ_(o) are mass attenuation coefficients for contrastagent and other organ tissues, which are known constants withλ_(c)>λ_(o). X_(c) and X_(o) are X-ray traveling distance in contrastagent, i.e., vessel, and other organ tissues.

The attenuated X-ray beams 102 then form angiographic projection image103 on the projection plane. Angiographic projection image 103 haspixels of different intensities. FIG. 1 illustrates how pixelintensities in angiographic projection image 103 are influenced by theX-ray travelling distance X_(c) in vessel. As shown in FIG. 1 , thepixel intensities are darker where the vessel shape is more along,instead of perpendicular to, the X-ray projection direction, due to thelarger X_(c). Intuitively, darker vessel intensities are shown on theprojected images where the vessel travels along the X-ray projectiondirection for a longer distance X_(c), as the X-ray is attenuated by athicker layer of contrast agent. Thus, reconstructing the depth andvessel shape from a single projection image is theoretically feasible.

FIG. 2 illustrates an exemplary image reconstruction system 200,according to some embodiments of the present disclosure. Consistent withthe present disclosure, image reconstruction system 200 may beconfigured to reconstruct a 3D image (also referred to as a 3D model)from a single-view 2D image acquired by an image acquisition device 205and optionally perform a diagnosis based on the reconstructed image.

In some embodiments, image acquisition device 205 may be a C-arm X-rayscanner used to acquire angiographic projection images. In some otherembodiments, image acquisition device 205 may be an imaging device thatacquires 2D images through projections. For example, image acquisitiondevice may use imaging modalities including, but are not limited to,Cone Beam CT (CBCT), Spiral CT, Positron Emission Tomography (PET),Single-Photon Emission Computed Tomography (SPECT), X-ray, opticaltomography, fluorescence imaging, and radiotherapy portal imaging, etc.,or the combination thereof.

As shown in FIG. 2 , image reconstruction system 200 may includecomponents for performing two phases, a training phase and a predictionphase. The prediction phase may also be referred to as an inferencephase. To perform the training phase, image reconstruction system 200may include a training database 201 and a model training device 202. Toperform the prediction phase, image reconstruction system 200 mayinclude an image processing device 203 and a medical image database 204.In some embodiments, image reconstruction system 200 may include more orless of the components shown in FIG. 2 . For example, when a learningmodel used for reconstructing the 3D images is pre-trained and provided,image reconstruction system 200 may include only image processing device203 and medical image database 204.

Image reconstruction system 200 may optionally include a network 206 tofacilitate the communication among the various components of imagereconstruction system 200, such as databases 201 and 204, devices 202,203, and 205. For example, network 206 may be a local area network(LAN), a wireless network, a cloud computing environment (e.g., softwareas a service, platform as a service, infrastructure as a service), aclient-server, a wide area network (WAN), etc. In some embodiments,network 206 may be replaced by wired data communication systems ordevices.

In some embodiments, the various components of image reconstructionsystem 200 may be remote from each other or in different locations andbe connected through network 206 as shown in FIG. 2 . In somealternative embodiments, certain components of image reconstructionsystem 200 may be located on the same site or inside one device. Forexample, training database 201 may be located on-site with or be part ofmodel training device 202. As another example, model training device 202and image processing device 203 may be inside the same computer orprocessing device.

Model training device 202 may use the training data received fromtraining database 201 to train a 3D information inference model fordetermining 3D information from a single-view 2D image received from,e.g., medical image database 204. In some embodiments, model trainingdevice 202 may train other learning models, such as an imagereconstruction model for reconstructing the 3D image from 3D informationdetermined by the inference model. As shown in FIG. 2 , model trainingdevice 202 may communicate with training database 201 to receive one ormore sets of training data. In certain embodiments, each set of trainingdata may include ground truth 3D information obtained through humanannotation and/or automatically computed by computers.

In some embodiments, the training phase may be performed “online” or“offline.” “Online” training refers to performing the training phasecontemporarily with the prediction phase, e.g., learning the model inreal-time just prior to analyzing a medical image. An “online” trainingmay have the benefit to obtain a most updated learning model based onthe training data that is then available. However, “online” training maybe computational costive to perform and may not always be possible ifthe training data is large and/or the model is complicated. Consistentwith the present disclosure, “offline” training is used where thetraining phase is performed separately from the prediction phase. Thelearned model trained offline is saved and reused for analyzing images.

Model training device 202 may be implemented with hardware speciallyprogrammed by software that performs the training process. For example,model training device 202 may include a processor and a non-transitorycomputer-readable medium (discussed in detail in connection with FIG. 3). The processor may conduct the training by performing instructions ofa training process stored in the computer-readable medium. Modeltraining device 202 may additionally include input and output interfacesto communicate with training database 201, network 206, and/or a userinterface (not shown). The user interface may be used for selecting setsof training data, adjusting one or more parameters of the trainingprocess, selecting or modifying a framework of the learning model,and/or manually or semi-automatically providing prediction resultsassociated with an image for training.

Consistent with some embodiments, the trained model may include avariety of modules or layers arranged in series and/or in parallel. Insome embodiments, the 3D information inference model may be implementedas a regression model trained with exemplar training data using deeplearning or other machine learning models.

Returning to FIG. 2 , the trained diagnosis model may be used by theimage processing device to reconstruct the 3D images for diagnosispurposes. Image processing device 203 may receive the trained models,e.g., the 3D information inference model and/or the image reconstructionmodel, from model training device 202. Image processing device 203 mayinclude a processor and a non-transitory computer-readable medium(discussed in detail in connection with FIG. 3 ). The processor mayperform instructions of a medical image diagnostic analysis programstored in the medium. Image processing device 203 may additionallyinclude input and output interfaces (discussed in detail in connectionwith FIG. 3 ) to communicate with medical image database 204, network206, and/or a user interface (not shown). The user interface may be usedfor selecting a single-view 2D image for reconstruction, initiating thereconstruction process, and displaying the reconstruction results.

Image processing device 203 may communicate with medical image database204 to receive single-view 2D images. The single-view 2D images may beprojection images of one or more 3D objects (e.g., a vessel) acquired byimage acquisition devices 205. Image processing device 203 mayreconstruct a 3D image of the 3D object from each single-view 2D image.In some embodiments, image processing device 203 may first determine 3Dinformation from the single-view 2D image using the trained inferencemodel. In some embodiments, the inferred 3D information can be depthinformation indicating a distance between each key point or dense pointof the 3D object and a projection plane of the single-view 2D image. Insome alternative embodiments, the inferred 3D information can be modelparameters such as shape parameters and/or pose parameters of adeformable model for the 3D object. Image processing device 203 may thenconstruct a 3D model of the 3D object using the inferred 3D information.Depending on the type of the 3D information, the reconstruction may be adepth-based reconstruction or a model-based reconstruction. The imagereconstruction process performed by image processing device 203 will bedescribed in more detail in connection with FIGS. 4, 5A-5B, and 6 .

Systems and methods disclosed in the present disclosure may beimplemented using a computer system, such as shown in FIG. 3 . In someembodiments, image processing device 203 may be a dedicated device or ageneral-purpose device. For example, the image processing device 203 maybe a computer customized for a hospital for processing image dataacquisition and image data processing tasks, or a server in a cloudenvironment. The image processing device 203 may include one or moreprocessor(s) 308 and one or more storage device(s) 304. The processor(s)308 and the storage device(s) 304 may be configured in a centralized ordistributed manner. The image processing device 203 may also include amedical database (optionally stored in storage device 304 or in a remotestorage), an input/output device (not shown, but which may include atouch screen, keyboard, mouse, speakers/microphone, or the like), anetwork interface such as communication interface 302, a display (notshown, but which may be a cathode ray tube (CRT) or liquid crystaldisplay (LCD) or the like), and other accessories or peripheral devices.The various elements of image processing device 203 may be connected bya bus 310, which may be a physical and/or logical bus in a computingdevice or among computing devices.

The processor 308 may be a processing device that includes one or moregeneral processing devices, such as a microprocessor, a centralprocessing unit (CPU), a graphics processing unit (GPU), and the like.More specifically, the processor 308 may be a complex instruction setcomputing (CISC) microprocessor, a reduced instruction set computing(RISC) microprocessor, a very long instruction word (VLIW)microprocessor, a processor running other instruction sets, or aprocessor that runs a combination of instruction sets. The processor 308may also be one or more dedicated processing devices such asapplication-specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), digital signal processors (DSPs), system-on-chip(SoCs), and the like.

The processor 308 may be communicatively coupled to the storage device304 and configured to execute computer-executable instructions storedtherein. For example, as illustrated in FIG. 3 , a bus 310 may be used,although a logical or physical star or ring topology would be examplesof other acceptable communication topologies. The storage device 304 mayinclude a read-only memory (ROM), a flash memory, random access memory(RAM), a static memory, a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, nonremovable, or other types ofstorage device or tangible (e.g., non-transitory) computer-readablemedium. In some embodiments, the storage device 304 may storecomputer-executable instructions of one or more processing programs anddata generated when a computer program is executed. The processor mayexecute the processing program to implement each step of the methodsdescribed below. The processor may also send/receive medical datato/from the storage device.

The image processing device 203 may also include one or more digitaland/or analog communication (input/output) devices, not illustrated inFIG. 3 . For example, the input/output device may include a keyboard anda mouse or trackball that allow a user to provide input. The imageprocessing device 203 may further include a network interface,illustrated as communication interface 302, such as a network adapter, acable connector, a serial connector, a USB connector, a parallelconnector, a high-speed data transmission adapter such as optical fiber,USB 3.0, lightning, a wireless network adapter such as a WiFi adapter,or a telecommunication (3G, 4G/LTE, etc.) adapter and the like. Theimage processing device 203 may be connected to a network through thenetwork interface. The image processing device 203 may further include adisplay, as mentioned above. In some embodiments, the display may be anydisplay device suitable for displaying a medical image and itssegmentation results. For example, the image display may be an LCD, aCRT, or an LED display.

The image processing device 203 may be connected to model trainingdevice 202 and image acquisition device 205 as discussed above withreference to FIG. 2 . Other implementations are also possible.

FIG. 4 illustrates an exemplary three-dimensional image reconstructionframework 400 (hereafter framework 400) for performing the single-viewthree-dimensional reconstruction according to an embodiment of thepresent disclosure. Framework 400 may be implemented by image processingdevice 203 by executing computer instructions loaded in its memory. Insome embodiments, framework 400 may contain two modules: a 3Dinformation inference module 410 and a 3D model generation module 420.3D information inference module 410 receives a single-view 2D image 401and infers 3D information 403 necessary to reconstruct the 3D model ofthe 3D object (e.g., a vessel). Again, single-view 2D image 401 may be aprojection image acquired in a single projection direction.

In some embodiments, framework 400 could be implemented for depth-basedreconstruction or model-based reconstruction, or a hybrid of thereof.For example, FIG. 5A illustrates an exemplary three-dimensional imagereconstruction framework 510 (hereafter framework 510) for performing asingle-view three-dimensional reconstruction using a depth-basedapproach according to an embodiment of the present disclosure, and FIG.5B illustrates an exemplary three-dimensional image reconstructionframework 520 (hereafter framework 520) for performing a single-viewthree-dimensional reconstruction using a model-based approach accordingto an embodiment of the present disclosure. FIGS. 4, 5A, and 5B will bedescribed together.

3D information inference module 410 may use image processing techniquesand analytical formula using Equation (1), or implemented as alearning-based model formulated as a regression problem trained withexemplar training data using deep learning or other machine learningtechniques. In some embodiments, the 3D information inference model cantake the 2D image acquisition meta information, manually craftedfeatures, image patches (2D image patches or 2D+time patches fromvideo), or the whole image/video as input. Depending on thereconstruction approach used, 3D information inference module 410 mayoutput different types of 3D information for later constructing the 3Dmodel.

For the depth-based reconstruction framework shown in FIG. 5A, the 3Dinformation 403 in FIG. 4 could be depth information 403A on certain keypoints or dense points. Depth information 403A includes the distancefrom a 3D point to the projected view image plane, for all pixels (densepoints) in the 2D single view image or key pixels (key points) of the 2Dsingle view image that are representative of the 3D object. Examples ofkey points may include landmarks of the 3D object or centerline points(e.g., for a vessel). In some embodiments, framework 510 may furtherinclude an optional key point detection module 511 for detecting thesekey points. For example, key point detection module 511 may obtain a 2Dcenterline and corresponding radii for centerline point by an automaticor manual segmentation of the target vessel from the 2D projectionimage. 3D information inference module 410A is then used to estimate thedepth on each centerline point. It should be noted that the centerlineextraction performed by key point detection module 511 is only optionaland not essential. Even without the centerline, depth information can beestimated densely for every pixel in the vessel or every pixel in thesingle-view 2D image.

For the model-based reconstruction framework shown in FIG. 5B, the 3Dinformation 403 in FIG. 4 could be model parameters 403B of model of the3D object. Accordingly, in this approach, 3D information inferencemodule 410B estimates the shape parameter which determines the shape,and the pose parameter which determines the projection relationship. Themodel of the 3D object may be a rigid or deformable model whose shape iscontrolled by a set of shape parameters, and projection specified bycorresponding pose parameters. In some embodiments, the model shapeparameters may be the shape variation mode weights specified by ontraining data during training of the inference model. For example,during training, the target object mean shape and shape variation modecan be obtained from training set by methods such as principal componentanalysis. Then the target object shape can be represented by a weightvector signaling the contribution of each shape variation mode. Forexample, a larger value of the first weight may indicate the object islonger along a certain axis, while a larger value of the second weightmay indicate a bigger bulge in the middle. Given the shape parametervector, a unique shaped model can be determined. The pose parametersdefine a projection relationship of the object (e.g., a blood vessel)with the predetermined projection direction. The pose parameters mayinclude, e.g., rotation and distance of the 3D model to the projectionview plane.

3D information inference module 410A may use an inference modelformulated to solve an optimization problem aimed at optimizing both theshape and pose parameters. 3D information inference module 410A solvesthis optimization problem so that the optimal shape and pose parametersare returned, whose simulated projection matches the given inputprojected view (the input 2D projection image) closely.

Returning back to FIG. 4 , 3D model generation module 420 can receivethe 3D information 403 and generate the 3D model based on the 3Dinformation 403, e.g., the estimated depth information (e.g., FIG. 5A)and/or the model shape and pose information (e.g., FIG. 5B). Fordepth-based reconstruction shown in FIG. 5A, the corresponding 3D modelgeneration module 420A reconstructs the 3D coordinates of the 3D modelbased on the (x, y) coordinates of each projected point in theprojection view plane and the corresponding depth, i.e., z coordinate,estimated by 3D information inference module 410A. For example, 3D modelgeneration module 420A can use the 2D centerline point coordinates, andcorresponding depth and radius to render a 3D vessel as a tube-likeobject. It should be noted that orthographic projection (aka parallelprojection) is assumed here but the method can be easily adapted andextended to perspective projection, in which the depth is along theprojection ray.

For model-based reconstruction shown in FIG. 5B, the corresponding 3Dmodel generation module 420B reconstructs the 3D model for the objectfrom the optimal set of shape parameters and pose parameters estimatedby 3D information inference module 410B. For example, the 3D model canbe constructed using the shape variation mode weights, the rotation anddistance of the model to the projection plane, etc.

In some embodiments, 3D model generation module 420 may further generatecorresponding projection parameters 405. Examples of projectionparameters 405 include rotation (the projection direction), distance(between the 3D object and the projection view plane), etc. The goal ofreconstruction is that projecting the reconstructed 3D model accordingto the corresponding projection parameters matches input single-view 2Dimage 401 as much as possible.

In some embodiments, the 3D model could be represented in differentforms, including a series of 3D centerline points with varyingdiameters, surface mesh or volumetric representation. The reconstructed3D model may be rendered and displayed on a display of image processingdevice 203 for a user to view. In some embodiments, the user caninteract with the 3D model, including adjusting the display view of the3D model, zoom-in/zoom-out the 3D model, or alter certain aspects of the3D model.

Various analyses and tasks can be performed, by image processing device203 or a separate device, on the reconstructed 3D image. For example,the image may be analyzed for a medical diagnosis of the patient. Insome embodiments, the analysis may calculate certain physiologicalparameters to aid the medical diagnosis. For example, when the 3D imageis a 3D model of a coronary artery, parameters such as a fractional flowreserve (FFR) value may be calculated for certain points of the bloodvessel. Based on the calculated FFR values, a medical diagnosisindicating the likelihood that the stenosis impedes oxygen delivery tothe heart muscle (myocardial ischemia) may be determined.

FIG. 6 shows a flowchart of an exemplary process 600 for performing athree-dimensional image reconstruction according to certain embodimentsof the present disclosure. Process 600 may be performed by imageprocessing device 203 using a three-dimensional image reconstructionframework, such as one shown in FIG. 4, 5A or 5B. As shown in FIG. 6 ,process 600 may include steps S602-S610. It is contemplated that process600 may include more or less steps as shown in FIG. 6 . In addition, thesteps may be performed in a sequential order or some steps may beperformed in parallel. The steps may also be performed in a differentorder as shown in FIG. 6 .

In step S602, a single-view 2D image capturing a 3D object is received.For example, the single-view 2D image may be an X-ray angiographic imageacquired by a C-arm X-ray scanner in a single projection direction. Insome embodiments, image processing device 203 may receive thesingle-view 2D image from a medical image database 204.

In step S604, a key points detection may be performed on the single-view2D image to identify key points of the 3D object. For example, when the3D object is a blood vessel, image processing device 203 may firstsegment the 2D image to obtain a centerline of the blood vessel and thenselect the key points on the centerline. Step S604 may be performed whenprocess 600 uses a depth-based reconstruction approach, so that depthinformation can be estimated for the key points (in step S606) to reducecomputational cost. Step S604 is optional so it can be skipped in someembodiments of process 600, where step S606 may estimate depthinformation for all pixels (dense points) in the 2D image.

In step S606, 3D information can be estimated from the single-view 2Dimage. Depending on the reconstruction approach used, image processingdevice 203 may estimate different types of 3D information. In thedepth-based approach, depth information associated with at least one keypoint or dense point of the blood vessel may be estimated, by using,e.g., 3D information inference module 410A. The depth information isindicative of a distance between each key point or dense point and aprojection plane of the single-view 2D image. In the model-basedapproach, model parameters such as shape parameters and/or poseparameters of a 3D target model of the object may be estimated, byusing, e.g., 3D information inference module 410B. The 3D target modelmay be a deformable model or a rigid model defined by the modelparameters. In some embodiments, step S606 may apply a deep learning ormachine learning model/network (e.g., an inference learning networkformulated as a regression problem) to perform the 3D informationinference. The inference model can be trained using training samples, aswill be described in detail in connection with FIG. 7 .

In step S608, the 3D model of the object may be reconstructed as a 3Dimage based on the estimated 3D information. In the depth-basedapproach, image processing device 203 may generate the 3D model based onthe (x, y) coordinates of the key point or dense point in the projectionplane of the single-view 2D image along with the depth information (zcoordinate) associated with the key point or dense point, by using,e.g., 3D model generation module 420A. In the model-based approach,image processing device 203 may generate the 3D model based on the modelparameters, by using, e.g., 3D model generation module 420B.

In step S610, the reconstructed 3D image may be provided for furtheranalysis and medical diagnosis. Physiological or medical parameters maybe calculated based on the 3D model of the object, and medical diagnosiscan be made based on the calculated parameters. In some embodiments,deep learning or other machine learning techniques can be used for themedical diagnosis from the 3D image.

FIG. 7 illustrates an exemplary process 700 for training athree-dimensional image reconstruction framework (e.g., framework 400),according to certain embodiments of the present disclosure. The trainingdata may include sample single-view images or videos 701 and theircorresponding 3D model projection annotations 702. 3D model projectionannotations 702 can be obtained in various ways. In some embodiments,another modality from which the 3D model can be readily obtained. Forexample, a 3D CT angiographic image can be acquired by imaging devicessuch a CT scanner, and the 3D model can be constructed from the acquired3D image. The projection parameters can be derived from geometricparameters recorded by the imaging acquisition device (e.g., an imagingscanner). These parameters can also be refined by optimizing thealignment of projected 3D model and angiographic images. In someembodiments, the 3D model projection annotation can be obtained usingmulti-view 3D model reconstruction algorithm. In some embodiments, the3D model projection annotation can also be synthetic data obtained byfirst rendering a 3D model and then projecting the 3D model to produce asynthetic single-view projection image using an imagegenerator/renderer. The synthetic data could be realistic given apowerful image generator/renderer. In yet some embodiments, humanannotator can finetune annotations of the 3D model and projectionparameters.

In step 710, a 3D information inference module (e.g., module 410) may betrained with the training data to infer the 3D information from a 2Dprojection image. The training of the 3D information inference model canbe conducted according to a predetermined output format (e.g., depthinformation or model parameters). For a depth-based system, the output3D information format is the depth, i.e., distance from 3D point to theprojected view image plane, for dense or key pixels such as centerlinein the single view projection image. In this case, the inference modelmay be formulated with physics-based formula computations, with certainparameters tuned based on the training data. Accordingly, a depthinformation inference module, such as module 410A, is trained. For adeformable model-based system, the output 3D information format is themodel shape parameters (such as the shape variation mode weightsspecified by the principal component analysis on training data) thatdefines the shape of the object model, and/or the pose parameters (suchas rotation and distance of the 3D model to the projection plane) thatindicates a projection relationship of the object with the predeterminedprojection direction. In some embodiments, for a model-based system, the3D information inference module may be a machine learning model or adeep learning model. For example, the learning model may be formulatedas a regression problem. Accordingly, a model parameter inferencemodule, such as module 410B, is trained. The training may use a costfunction that optimize the depth information or set of shape/poseparameters such that a simulated projection image obtained by projectingthe 3D model along a predetermined projection direction matches theground truth projection image along that same projection direction.

In step 720, a 3D model generation module (e.g., module 420) may bedeveloped for generating a 3D model of the object based on the 3Dinformation. The 3D model generation module may be developed accordingto the 3D information inference output format. For a depth-based system,the output 3D information format is the depth, therefore thecorresponding 3D model generation module is developed to reconstruct the3D coordinates and model based on the (x, y) coordinate of eachprojected point and the corresponding depth, i.e., z coordinate.Accordingly, a depth-based 3D model generation module, such as module410B, is developed. For a deformable model-based system, the 3Dinformation format is the model shape parameters and/or pose parameters,therefore the corresponding 3D model generation module is developed toreconstruct the 3D model from the shape parameters and pose parameters.For example, a model-based 3D model generation module, such as module410B, is developed.

In some embodiments, the computer-readable medium may include volatileor non-volatile, magnetic, semiconductor, tape, optical, removable,non-removable, or other types of computer-readable medium orcomputer-readable storage devices. For example, the computer-readablemedium may be the storage device or the memory module having thecomputer instructions stored thereon, as disclosed. In some embodiments,the computer-readable medium may be a disc or a flash drive having thecomputer instructions stored thereon.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed system andrelated methods. Other embodiments will be apparent to those skilled inthe art from consideration of the specification and practice of thedisclosed system and related methods.

For example, although angiographic images of vessel (such as coronaryartery, neck artery and brain artery images) are used as an example fordisclosing the systems and methods herein, it is contemplated that thedisclosed systems and methods can be adapted and applied to otherpotential applications. The disclosed systems and methods can be used toreconstruct 3D images of any 3D objects that can be captured byprojection imaging, beyond just vessels. For example, the disclosedsystems and methods such as reconstructing chest organs from chestX-rays. Such adaption and application are within the ability of anordinary skill in art. Therefore, the scope of the disclosure should notbe construed to be limited to reconstructing blood vessel models, butencompass reconstruction of other three-dimensional biomedical imagesfrom a single-view projection image using the disclosed technique.

Further, the disclosed systems and methods can also be used toreconstruct 3D images of any imaging modality that can obtain projectionimages, beyond just X-ray or CT. For example, imaging modalities in thedisclosed systems and methods may be alternatively or additionallyapplied to other imaging modalities where the pixel intensity varieswith the distance traveled by imaging particles, such as CT, cone beamcomputed tomography (CBCT), Spiral CT, positron emission tomography(PET), single-photon emission computed tomography (SPECT), etc.

It is intended that the specification and examples be considered asexemplary only, with a true scope being indicated by the followingclaims and their equivalents.

What is claimed is:
 1. A device for three-dimensional blood vesselreconstruction, comprising: an interface configured to receive onesingle-view two-dimensional image of a blood vessel of a patient,wherein the single-view two-dimensional image is a projection imageacquired in a predetermined projection direction; and a processorconfigured to: estimate three-dimensional information of the bloodvessel from the one single-view two-dimensional image using an inferencemodel, wherein the inference model is trained using training dataincluding sample single-view images and their corresponding 3D modelprojection annotations; and reconstruct a three-dimensional model of theblood vessel based on the three-dimensional information.
 2. The deviceaccording to claim 1, wherein the three-dimensional informationestimated by the inference model comprises depth information associatedwith at least one key point or dense point of the blood vessel, whereinthe depth information is indicative of a distance between each key pointor dense point and a projection plane of the single-view two-dimensionalimage.
 3. The device according to claim 1, wherein the three-dimensionalinformation estimated by the inference model comprises at least oneshape parameter defining a shape of a model of the blood vessel.
 4. Thedevice according to claim 3, wherein the model of the blood vessel is arigid model or a deformable model of the blood vessel.
 5. The deviceaccording to claim 3, wherein the three-dimensional informationestimated by the inference model further comprises at least one poseparameter indicative of a projection relationship of the blood vesselwith the predetermined projection direction.
 6. The device according toclaim 1, wherein the single-view two-dimensional image is an X-rayangiographic image of the patient acquired by a C-arm x-ray scanner. 7.A computer-implemented method for performing three-dimensional imagereconstruction, comprising: receiving one single-view two-dimensionalimage of a patient, wherein the single-view two-dimensional image is aprojection image acquired in a predetermined projection direction;estimating, by a processor, three-dimensional information from the onesingle-view two-dimensional image using an inference model, whereininference model is trained using training data including samplesingle-view images and their corresponding 3D model projectionannotations; and reconstructing, by the processor, a three-dimensionalmodel based on the three-dimensional information.
 8. Thecomputer-implemented method according to claim 7, wherein thesingle-view two-dimensional image captures a blood vessel of thepatient, and the three-dimensional model is a three-dimensional model ofthe blood vessel.
 9. The computer-implemented method according to claim8, wherein the three-dimensional information estimated by the inferencemodel comprises depth information associated with at least one key pointor dense point of the blood vessel, wherein the depth information isindicative of a distance between each key point or dense point and aprojection plane of the single-view two-dimensional image.
 10. Thecomputer-implemented method according to claim 9, further comprising:determine a centerline of the blood vessel from the single viewtwo-dimensional image; determine at least one point on the centerline asthe at least one key point or dense point of the blood vessel; and applythe inference model to determine the depth information of the at leastone key point or dense point of the blood vessel.
 11. Thecomputer-implemented method according to claim 9, wherein reconstructinga three-dimensional model based on the three-dimensional informationfurther comprises: constructing the three-dimensional model of the bloodvessel using coordinates of the key point or dense point of the bloodvessel in the projection plane and the depth information of the keypoint or dense point.
 12. The computer-implemented method according toclaim 8, wherein the three-dimensional information estimated by theinference model comprises at least one shape parameter defining a shapeof a model of the blood vessel.
 13. The computer-implemented methodaccording to claim 12, wherein the model of the blood vessel is a rigidmodel or a deformable model of the blood vessel.
 14. Thecomputer-implemented method according to claim 12, wherein the at leastone shape parameter comprises shape variation mode weights defining atarget object shape determined from training data used for training theinference model.
 15. The computer-implemented method according to claim12, wherein the three-dimensional information estimated by the inferencemodel further comprises at least one pose parameter indicative of aprojection relationship of the blood vessel with the predeterminedprojection direction.
 16. The computer-implemented method according toclaim 15, wherein reconstructing a three-dimensional model based on thethree-dimensional information further comprises: constructing thethree-dimensional model of the blood vessel using the at least one shapeparameter and the at least one pose parameter.
 17. Thecomputer-implemented method according to claim 7, wherein thesingle-view two-dimensional image is an X-ray angiographic image of thepatient.
 18. A non-transitory computer-readable medium, havinginstructions stored thereon, wherein the instructions, when executed bya processor, perform a method for performing three-dimensional imagereconstruction, wherein the method comprises: receiving one single-viewtwo-dimensional image of a patient, wherein the single-viewtwo-dimensional image is a projection image acquired in a predeterminedprojection direction; estimating three-dimensional information from theone single-view two-dimensional image using an inference model, whereinthe inference model is trained using training data including samplesingle-view images and their corresponding 3D model projectionannotations; and reconstructing a three-dimensional model based on thethree-dimensional information.